Frequentist Inference Case Study - Part A

1. Learning objectives

Welcome to part A of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. Frequentist inference is the process of deriving conclusions about an underlying distribution via the observation of data. In particular, you'll practice writing Python code to apply the following statistical concepts:

  • the z-statistic z 動画
  • the t-statistic t 動画
  • the difference and relationship between the two
  • the Central Limit Theorem, including its assumptions and consequences
  • how to estimate the population mean and standard deviation from a sample
  • the concept of a sampling distribution of a test statistic, particularly for the mean

  • how to combine these concepts to calculate a confidence interval

  • test procedure

Prerequisites

To be able to complete this notebook, you are expected to have a basic understanding of:

  • what a random variable is (p.400 of Professor Spiegelhalter's The Art of Statistics, hereinafter AoS)
  • what a population, and a population distribution, are (p. 397 of AoS)
  • a high-level sense of what the normal distribution is (p. 394 of AoS)
  • what the t-statistic is (p. 275 of AoS)

Happily, these should all be concepts with which you are reasonably familiar after having read ten chapters of Professor Spiegelhalter's book, The Art of Statistics.

We'll try to relate the concepts in this case study back to page numbers in The Art of Statistics so that you can focus on the Python aspects of this case study. The second part (part B) of this case study will involve another, more real-world application of these tools.

For this notebook, we will use data sampled from a known normal distribution. This allows us to compare our results with theoretical expectations.

2. An introduction to sampling from the normal distribution

First, let's explore the ways we can generate the normal distribution. While there's a fair amount of interest in sklearn within the machine learning community, you're likely to have heard of scipy if you're coming from the sciences. For this assignment, you'll use scipy.stats to complete your work.

This assignment will require some digging around and getting your hands dirty (your learning is maximized that way)! You should have the research skills and the tenacity to do these tasks independently, but if you struggle, reach out to your immediate community and your mentor for help.

In [1]:
from scipy.stats import norm
from scipy.stats import t
import numpy as np
import pandas as pd
from numpy.random import seed
import matplotlib.pyplot as plt

Q1: Call up the documentation for the norm function imported above. (Hint: that documentation is here). What is the second listed method?

A:

It's pdf(x, loc=0, scale=1), Probability density function.

Q2: Use the method that generates random variates to draw five samples from the standard normal distribution.

A:

In [2]:
seed(47)
# draw five samples here

r = norm.rvs(size=5)

plt.plot(r,marker='.',linestyle='none');

Q3: What is the mean of this sample? Is it exactly equal to the value you expected? Hint: the sample was drawn from the standard normal distribution. If you want a reminder of the properties of this distribution, check out p. 85 of AoS.

A:

In [3]:
# Calculate and print the mean here, hint: use np.mean()
mean = np.mean(r)
mean
Out[3]:
0.19355593334131074

Q4: What is the standard deviation of these numbers? Calculate this manually here as $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}$ (This is just the definition of standard deviation given by Professor Spiegelhalter on p.403 of AoS). Hint: np.sqrt() and np.sum() will be useful here and remember that numPy supports broadcasting.

A:

In [4]:
np.sqrt(np.sum((r-mean)**2)/ len(r))
Out[4]:
0.9606195639478641
In [5]:
np.std(r)
Out[5]:
0.9606195639478641
In [6]:
# reference

r.std(
#     ddof=0
)
Out[6]:
0.9606195639478641

Here we have calculated the actual standard deviation of a small data set (of size 5). But in this case, this small data set is actually a sample from our larger (infinite) population. In this case, the population is infinite because we could keep drawing our normal random variates until our computers die!

In general, the sample mean we calculate will not be equal to the population mean (as we saw above). A consequence of this is that the sum of squares of the deviations from the population mean will be bigger than the sum of squares of the deviations from the sample mean. In other words, the sum of squares of the deviations from the sample mean is too small to give an unbiased estimate of the population variance. An example of this effect is given here. Scaling our estimate of the variance by the factor $n/(n-1)$ gives an unbiased estimator of the population variance. This factor is known as Bessel's correction. The consequence of this is that the $n$ in the denominator is replaced by $n-1$.

You can see Bessel's correction reflected in Professor Spiegelhalter's definition of variance on p. 405 of AoS.

Q5: If all we had to go on was our five samples, what would be our best estimate of the population standard deviation? Use Bessel's correction ($n-1$ in the denominator), thus $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}}$.

A:

In [7]:
np.sqrt(np.sum((r-mean)**2)/ (len(r)-1))
Out[7]:
1.0740053227518152
In [8]:
# reference
r.std(ddof=1)
Out[8]:
1.0740053227518152

Q6: Now use numpy's std function to calculate the standard deviation of our random samples. Which of the above standard deviations did it return?

A:

In [9]:
np.std(r)
Out[9]:
0.9606195639478641

Numpy std function has returned the former one.

Q7: Consult the documentation for np.std() to see how to apply the correction for estimating the population parameter and verify this produces the expected result.

A:

In [10]:
np.std(r), np.std(r,ddof=1)
Out[10]:
(0.9606195639478641, 1.0740053227518152)

The two values did not exactly match, which concludes that the sample is not unbiased.

Summary of section

In this section, you've been introduced to the scipy.stats package and used it to draw a small sample from the standard normal distribution. You've

  • calculated the average (the mean) of this sample and
  • seen that this is not exactly equal to the expected population parameter (which we know because we're generating the random variates from a specific, known distribution). You've been introduced to two ways of calculating the standard deviation; one uses $n$ in the denominator and the other uses $n-1$ (Bessel's correction). You've also seen which of these calculations np.std() performs by default and how to get it to generate the other.

You use $n$ as the denominator if you want to calculate the standard deviation of a sequence of numbers. You use $n-1$ if you are using this sequence of numbers to estimate the population parameter. This brings us to some terminology that can be a little confusing.

The population parameter is traditionally written as $\sigma$ and the sample statistic as $s$. Rather unhelpfully, $s$ is also called the sample standard deviation (using $n-1$) whereas the standard deviation of the sample uses $n$. That's right, we have the sample standard deviation and the standard deviation of the sample and they're not the same thing!

The sample standard deviation \begin{equation} s = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}} \approx \sigma, \end{equation} is our best (unbiased) estimate of the population parameter ($\sigma$).

If your dataset is your entire population, you simply want to calculate the population parameter, $\sigma$, via \begin{equation} \sigma = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}} \end{equation} as you have complete, full knowledge of your population. In other words, your sample is your population. It's worth noting that we're dealing with what Professor Spiegehalter describes on p. 92 of AoS as a metaphorical population: we have all the data, and we act as if the data-point is taken from a population at random. We can think of this population as an imaginary space of possibilities.

If, however, you have sampled from your population, you only have partial knowledge of the state of your population. In this case, the standard deviation of your sample is not an unbiased estimate of the standard deviation of the population, in which case you seek to estimate that population parameter via the sample standard deviation, which uses the $n-1$ denominator.

Great work so far! Now let's dive deeper.

3. Sampling distributions

So far we've been dealing with the concept of taking a sample from a population to infer the population parameters. One statistic we calculated for a sample was the mean. As our samples will be expected to vary from one draw to another, so will our sample statistics. If we were to perform repeat draws of size $n$ and calculate the mean of each, we would expect to obtain a distribution of values. This is the sampling distribution of the mean. The Central Limit Theorem (CLT) tells us that such a distribution will approach a normal distribution as $n$ increases (the intuitions behind the CLT are covered in full on p. 236 of AoS). For the sampling distribution of the mean, the standard deviation of this distribution is given by

\begin{equation} \sigma_{mean} = \frac{\sigma}{\sqrt n} \end{equation}

where $\sigma_{mean}$ is the standard deviation of the sampling distribution of the mean and $\sigma$ is the standard deviation of the population (the population parameter).

This is important because typically we are dealing with samples from populations and all we know about the population is what we see in the sample. From this sample, we want to make inferences about the population. We may do this, for example, by looking at the histogram of the values and by calculating the mean and standard deviation (as estimates of the population parameters), and so we are intrinsically interested in how these quantities vary across samples.

In other words, now that we've taken one sample of size $n$ and made some claims about the general population, what if we were to take another sample of size $n$? Would we get the same result? Would we make the same claims about the general population? This brings us to a fundamental question: when we make some inference about a population based on our sample, how confident can we be that we've got it 'right'?

We need to think about estimates and confidence intervals: those concepts covered in Chapter 7, p. 189, of AoS.

Now, the standard normal distribution (with its variance equal to its standard deviation of one) would not be a great illustration of a key point. Instead, let's imagine we live in a town of 50,000 people and we know the height of everyone in this town. We will have 50,000 numbers that tell us everything about our population. We'll simulate these numbers now and put ourselves in one particular town, called 'town 47', where the population mean height is 172 cm and population standard deviation is 5 cm.

In [11]:
seed(47)
pop_heights = norm.rvs(172, 5, size=50000)
In [12]:
_ = plt.hist(pop_heights, bins=30)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(172, color='r')
_ = plt.axvline(172+5, color='r', linestyle='--')
_ = plt.axvline(172-5, color='r', linestyle='--')
_ = plt.axvline(172+10, color='r', linestyle='-.')
_ = plt.axvline(172-10, color='r', linestyle='-.')

Now, 50,000 people is rather a lot to chase after with a tape measure. If all you want to know is the average height of the townsfolk, then can you just go out and measure a sample to get a pretty good estimate of the average height?

In [13]:
def townsfolk_sampler(n):
    return np.random.choice(pop_heights, n)

Let's say you go out one day and randomly sample 10 people to measure.

In [14]:
seed(47)
daily_sample1 = townsfolk_sampler(10)
In [15]:
_ = plt.hist(daily_sample1, bins=10)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in sample size 10')

The sample distribution doesn't resemble what we take the population distribution to be. What do we get for the mean?

In [16]:
np.mean(daily_sample1)
Out[16]:
173.47911444163503

And if we went out and repeated this experiment?

In [17]:
daily_sample2 = townsfolk_sampler(10)
In [18]:
np.mean(daily_sample2)
Out[18]:
173.7317666636263

Q8: Simulate performing this random trial every day for a year, calculating the mean of each daily sample of 10, and plot the resultant sampling distribution of the mean.

A:

In [ ]:
In [19]:
seed(47)
# take your samples here
ls = []
for day in range(365):
    daily_sample = townsfolk_sampler(10)
    mean = np.mean(daily_sample)
    ls.append(mean)
In [20]:
_ = plt.hist(ls, bins='auto')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('count of day mean')
_ = plt.title('Distribution of heights')

The above is the distribution of the means of samples of size 10 taken from our population. The Central Limit Theorem tells us the expected mean of this distribution will be equal to the population mean, and standard deviation will be $\sigma / \sqrt n$, which, in this case, should be approximately 1.58.

?? which n is it?

Q9: Verify the above results from the CLT.

A:

In [21]:
np.mean(ls), 172, np.std(ls),5/np.sqrt(10)
Out[21]:
(171.8660049358649, 172, 1.5756704135286475, 1.5811388300841895)

Remember, in this instance, we knew our population parameters, that the average height really is 172 cm and the standard deviation is 5 cm, and we see some of our daily estimates of the population mean were as low as around 168 and some as high as 176.

Q10: Repeat the above year's worth of samples but for a sample size of 50 (perhaps you had a bigger budget for conducting surveys that year)! Would you expect your distribution of sample means to be wider (more variable) or narrower (more consistent)? Compare your resultant summary statistics to those predicted by the CLT.

A:

In [22]:
seed(47)
# calculate daily means from the larger sample size here
ls = []
for day in range(365):
    daily_sample = townsfolk_sampler(50)
    mean = np.mean(daily_sample)
    ls.append(mean)
In [23]:
_ = plt.hist(ls, bins='auto')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('count of day mean')
_ = plt.title('Distribution of heights')
In [24]:
np.mean(ls), 172, np.std(ls),5/np.sqrt(50)
Out[24]:
(171.94366080916114, 172, 0.6736107539771146, 0.7071067811865475)

What we've seen so far, then, is that we can estimate population parameters from a sample from the population, and that samples have their own distributions. Furthermore, the larger the sample size, the narrower are those sampling distributions.

Normally testing time!

All of the above is well and good. We've been sampling from a population we know is normally distributed, we've come to understand when to use $n$ and when to use $n-1$ in the denominator to calculate the spread of a distribution, and we've seen the Central Limit Theorem in action for a sampling distribution. All seems very well behaved in Frequentist land. But, well, why should we really care?

Remember, we rarely (if ever) actually know our population parameters but we still have to estimate them somehow. If we want to make inferences to conclusions like "this observation is unusual" or "my population mean has changed" then we need to have some idea of what the underlying distribution is so we can calculate relevant probabilities. In frequentist inference, we use the formulae above to deduce these population parameters. Take a moment in the next part of this assignment to refresh your understanding of how these probabilities work.

Recall some basic properties of the standard normal distribution, such as that about 68% of observations are within plus or minus 1 standard deviation of the mean. Check out the precise definition of a normal distribution on p. 394 of AoS.

Q11: Using this fact, calculate the probability of observing the value 1 or less in a single observation from the standard normal distribution. Hint: you may find it helpful to sketch the standard normal distribution (the familiar bell shape) and mark the number of standard deviations from the mean on the x-axis and shade the regions of the curve that contain certain percentages of the population.

A:

Calculating this probability involved calculating the area under the curve from the value of 1 and below. To put it in mathematical terms, we need to integrate the probability density function. We could just add together the known areas of chunks (from -Inf to 0 and then 0 to $+\sigma$ in the example above). One way to do this is to look up tables (literally). Fortunately, scipy has this functionality built in with the cdf() function.

In [25]:
np.sum(pop_heights<=172+5)/len(pop_heights)
Out[25]:
0.84002

50 + 34.1 = 84.1 %

Error function

Q12: Use the cdf() function to answer the question above again and verify you get the same answer.

A:

In [26]:
cdf = norm.cdf(1)
cdf
Out[26]:
0.8413447460685429

Q13: Using our knowledge of the population parameters for our townsfolks' heights, what is the probability of selecting one person at random and their height being 177 cm or less? Calculate this using both of the approaches given above.

A:

In [27]:
# sum of pop
np.sum(pop_heights<=177)/len(pop_heights)
Out[27]:
0.84002

calculating the area under the curve from the value of z

z = (177-172)/5 = 1

area = 50 + 34.1 = 84.1

In [28]:
# Cumulative Desity Function
norm.cdf((177-172)/5)
Out[28]:
0.8413447460685429

Q14: Turning this question around — suppose we randomly pick one person and measure their height and find they are 2.00 m tall. How surprised should we be at this result, given what we know about the population distribution? In other words, how likely would it be to obtain a value at least as extreme as this? Express this as a probability.

A:

In [29]:
np.sum((pop_heights>=199.6) & (pop_heights<200.6))/len(pop_heights)
Out[29]:
0.0
In [30]:
1-norm.cdf((200-172)/5)
Out[30]:
1.0717590259723409e-08

What we've just done is calculate the p-value of the observation of someone 2.00m tall (review p-values if you need to on p. 399 of AoS). We could calculate this probability by virtue of knowing the population parameters. We were then able to use the known properties of the relevant normal distribution to calculate the probability of observing a value at least as extreme as our test value.

We're about to come to a pinch, though. We've said a couple of times that we rarely, if ever, know the true population parameters; we have to estimate them from our sample and we cannot even begin to estimate the standard deviation from a single observation.

This is very true and usually we have sample sizes larger than one. This means we can calculate the mean of the sample as our best estimate of the population mean and the standard deviation as our best estimate of the population standard deviation.

In other words, we are now coming to deal with the sampling distributions we mentioned above as we are generally concerned with the properties of the sample means we obtain.

Above, we highlighted one result from the CLT, whereby the sampling distribution (of the mean) becomes narrower and narrower with the square root of the sample size. We remind ourselves that another result from the CLT is that even if the underlying population distribution is not normal, the sampling distribution will tend to become normal with sufficiently large sample size. (Check out p. 199 of AoS if you need to revise this). This is the key driver for us 'requiring' a certain sample size, for example you may frequently see a minimum sample size of 30 stated in many places. In reality this is simply a rule of thumb; if the underlying distribution is approximately normal then your sampling distribution will already be pretty normal, but if the underlying distribution is heavily skewed then you'd want to increase your sample size.

Q15: Let's now start from the position of knowing nothing about the heights of people in our town.

  • Use the random seed of 47, to randomly sample the heights of 50 townsfolk
  • Estimate the population mean using np.mean
  • Estimate the population standard deviation using np.std (remember which denominator to use!)
  • Calculate the (95%) margin of error (use the exact critial z value to 2 decimal places - look this up or use norm.ppf()) Recall that the margin of error is mentioned on p. 189 of the AoS and discussed in depth in that chapter).
  • Calculate the 95% Confidence Interval of the mean (confidence intervals are defined on p. 385 of AoS)
  • Does this interval include the true population mean?

A:

In [31]:
seed(47)
# take your sample now
random_folk = townsfolk_sampler(50)


#  population mean, population std
print('(population mean, population std) =')
np.mean(pop_heights),np.std(pop_heights,ddof=0)

(population mean, population std) =
Out[31]:
(172.0192602425845, 4.990839139566387)
In [32]:
# the (95%) margin of error  (== MOE)


# 97.5% percentile, 
z = np.round(norm.ppf(0.975),2)

# population mean
population_m = np.mean(pop_heights)

# population std
population_std = np.std(pop_heights,ddof=0)


# sample size
sampleSize = len(random_folk)

MOEz = z * population_std/np.sqrt(sampleSize)


MOEz

print(f'MOE is {MOEz}')

MOE is 1.3833900301642617
In [33]:
# sample mean
m = np.mean(random_folk)

_ = plt.hist(random_folk, bins='auto')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(m, color='b')
_ = plt.axvline(m+MOEz, color='b', linestyle='--')
_ = plt.axvline(m-MOEz, color='b', linestyle='--')
_ = plt.axvline(172, color='r')


print('H0 : sample mean == population mean,\
      \nH1: sample mean != population mean,\
      \nalpha=0.05\n')




print('z-test')
print(f'z value = {(m-population_m) / (population_std/np.sqrt(sampleSize))}')
print(f'z0 = {z}')
      
      
print(f'\nMOE with z-test is {round(MOEz,2)}cm')

print(f'\nConfidence intervals for the mean: (lower,upper) = \n{m-MOEz,m+MOEz}')

print(f'\nIs the population mean, 172cm, within the CI? => {m-MOEz < 172 and 172 < m+MOEz}')

H0 : sample mean == population mean,      
H1: sample mean != population mean,      
alpha=0.05

z-test
z value = 1.0799638373911342
z0 = 1.96

MOE with z-test is 1.38cm

Confidence intervals for the mean: (lower,upper) = 
(171.39812082751453, 174.16490088784306)

Is the population mean, 172cm, within the CI? => True

Q16: Above, we calculated the confidence interval using the critical z value. What is the problem with this? What requirement, or requirements, are we (strictly) failing?

A:

Requirement for z value is that we have to know the population parameters such as true population distribution. However, we usually do not know that before the calcualtion. This is the problem when we are using critical z value.

The requirement we are failing is that the sample size of 50 has not been confirmed to satisfify the requirement mentioned above. The sample size should have been enough to satisfy that Confidence interval includes true population mean.

Q17: Calculate the 95% confidence interval for the mean using the t distribution. Is this wider or narrower than that based on the normal distribution above? If you're unsure, you may find this resource resouceuseful. For calculating the critical value, remember how you could calculate this for the normal distribution using norm.ppf().

A:

In [36]:
# Calculate the 95% Confidence Interval of the mean (confidence intervals are defined on p. 385 of AoS)
# (== CI)

# 100% - 95% = 5%
alpha = 0.05

# degree of freedom
dof = sampleSize - 1

# probability point function for t
t_ = t.ppf(1-alpha/2,dof)
# ref: 
# https://www.statisticshowto.com/probability-and-statistics/confidence-interval/

# Unbiased estimate of population variance
# (sample standard deviation)
sampleStd = np.std(random_folk,ddof=1)

# Standard Error
SE = sampleStd/np.sqrt(sampleSize)

# CI with sample std
MOEt = t_ * SE


# print(sampleStd,population_std)




print('H0 : sample mean == population mean,\
      \nH1: sample mean != population mean,\
      \nalpha=0.05\n')

# Unbiased estimate of population variance
# (sample standard deviation)
print(f'Unbiased estimate of population variance = {sampleStd}')
print(f'Satandard Error = {SE}')

print('\nt-test')
print(f't value = {(m-population_m) / SE}')
print(f't0 = {t_}')
      
      

print(f'\nMOE with t-test is {round(MOEt,2)}cm')


print(f'\nConfidence intervals for the mean: (lower,upper) = \n{m-MOEt,m+MOEt}')


print(f'\nIs the population mean, 172cm, within the CI? => {m-MOEt < 172 and 172 < m+MOEt}')

H0 : sample mean == population mean,      
H1: sample mean != population mean,      
alpha=0.05

Unbiased estimate of population variance = 4.195424364433547
Satandard Error = 0.5933226036092445

t-test
t value = 1.2847152804519006
t0 = 2.009575234489209

MOE with t-test is 1.19cm

Confidence intervals for the mean: (lower,upper) = 
(171.589184447403, 173.9738372679546)

Is the population mean, 172cm, within the CI? => True
In [35]:
# sample mean
m = np.mean(random_folk)

_ = plt.hist(random_folk, bins='auto')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(m, color='b')
_ = plt.axvline(m+MOEt, color='b', linestyle='--')
_ = plt.axvline(m-MOEt, color='b', linestyle='--')
_ = plt.axvline(172, color='r')
print(f'\nIs the population mean, 172cm, within the CI? => {m-MOEt < 172 and 172 < m+MOEt}')

Is the population mean, 172cm, within the CI? => True

For the same 95% confidence criteria, t distribution has wider interval than the normal distribution (z distribution) due to the degree of freedom (dof).

However, actual sample distribution based on the t distribution and standard error (SE) has narrower interval because of small sample standard deviation (sampleStd).

===The end of my answer===

In [ ]:
**memo

This is slightly wider than the previous confidence interval. This reflects the greater uncertainty given that we are estimating population parameters from a sample.

4. Learning outcomes

Having completed this project notebook, you now have hands-on experience:

  • sampling and calculating probabilities from a normal distribution
  • identifying the correct way to estimate the standard deviation of a population (the population parameter) from a sample
  • with sampling distribution and now know how the Central Limit Theorem applies
  • with how to calculate critical values and confidence intervals